Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Metricbeat] Adding cloudwatch metricset into AWS module #11798

Merged
merged 17 commits into from
May 1, 2019
Merged

[Metricbeat] Adding cloudwatch metricset into AWS module #11798

merged 17 commits into from
May 1, 2019

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Apr 12, 2019

This PR is to add cloudwatch metricset into aws module for metricbeat. cloudwatch metricset will read target namespace from config file and query cloudwatch API ListMetrics and GetMetricData to get monitoring metrics for that specific namespace.

Currently the config for cloudwatch metric looks like:

  cloudwatch_metrics:
    - namespace: AWS/EC2
       metricname: CPUUtilization
       dimensions:
         - name: InstanceId
           value: i-0686946e22cf9494a
    - namespace: AWS/EC2
       metricname: DiskWriteBytes
       dimensions:
         - name: InstanceId
           value: i-0686946e22cf9494a
    - namespace: AWS/EBS
    - namespace: AWS/ELB

Fetch function starts with checking config to see how many listMetrics entries are there as well as namespaces.

- namespace: AWS/EC2
   metricname: CPUUtilization
   dimensions:
     - name: InstanceId
       value: i-0686946e22cf9494a

This count as one listMetrics entry. readCloudwatchConfig reads in the config and stores each listMetrics entry and namespace if there's no metricname information but only namespace, like

- namespace: AWS/ELB

For listMetricsTotal, this is already good for createEvents to use. But for namespacesTotal, we need to loop through each namespace and each region making GetListMetrics api call to get listMetricsOutput and use that for createEvents function.

In createEvents, there is a variable name identifier, which is the combination of all dimensions in listMetricsOutput. For example,

    - namespace: AWS/RDS
      metricname: CommitThroughput
      dimensions:
        - name: DBClusterIdentifier
          value: test1-cluster
        - name: Role
          value: READER

the identifier name is: DBClusterIdentifier,Role
the identifier value is: test1-cluster,READER

Some entries of listMetricsOuput does not have dimensions and that's why there is a eventsNoIdentifier to store events that came from listMetricsOutput that doesn't have dimensions/identifiers.

Please see more details in #11734

@kaiyan-sheng kaiyan-sheng self-assigned this Apr 12, 2019
@kaiyan-sheng kaiyan-sheng added Team:Integrations Label for the Integrations team in progress Pull request is currently in progress. [zube]: In Progress labels Apr 12, 2019
@kaiyan-sheng kaiyan-sheng marked this pull request as ready for review April 17, 2019 20:12
@kaiyan-sheng kaiyan-sheng requested review from a team as code owners April 17, 2019 20:12
@kaiyan-sheng kaiyan-sheng added review and removed [zube]: In Progress in progress Pull request is currently in progress. labels Apr 18, 2019
Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could take more advantage of the config parsing to simplify the code.

Also I am not sure if cloudwatch is generic enough to add its fields to the common structs for AWS config and metricset.

x-pack/metricbeat/module/aws/aws.go Outdated Show resolved Hide resolved
x-pack/metricbeat/module/aws/aws.go Outdated Show resolved Hide resolved
x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go Outdated Show resolved Hide resolved
x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go Outdated Show resolved Hide resolved
x-pack/metricbeat/module/aws/aws.go Outdated Show resolved Hide resolved
x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go Outdated Show resolved Hide resolved
x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go Outdated Show resolved Hide resolved
x-pack/metricbeat/module/aws/aws.go Outdated Show resolved Hide resolved
x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go Outdated Show resolved Hide resolved
@kaiyan-sheng
Copy link
Contributor Author

I think we could take more advantage of the config parsing to simplify the code.

Also I am not sure if cloudwatch is generic enough to add its fields to the common structs for AWS config and metricset.

@jsoriano Yes you are right, cloudwatch should not be a part of aws module config. I will move it under cloudwatch metricset. Thank you!

@kaiyan-sheng kaiyan-sheng added in progress Pull request is currently in progress. and removed review labels Apr 23, 2019
@jsoriano jsoriano dismissed their stale review April 24, 2019 10:30

Most changed addressed, I will review again.

@kaiyan-sheng
Copy link
Contributor Author

@jsoriano Thanks for the review and all the comments. Regarding to your comment about GetStartTimeEndTime, I will post a separate PR to address it since it's a function that used by all metricsets inside aws. Thank you!

@kaiyan-sheng kaiyan-sheng added review and removed in progress Pull request is currently in progress. labels Apr 25, 2019
Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I have added a couple of minor comments.

x-pack/metricbeat/module/aws/ec2/ec2_test.go Outdated Show resolved Hide resolved
x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go Outdated Show resolved Hide resolved
@exekias
Copy link
Contributor

exekias commented Apr 25, 2019

Could you please add a changelog entry?

@kaiyan-sheng kaiyan-sheng merged commit 55e9628 into elastic:master May 1, 2019
@kaiyan-sheng kaiyan-sheng deleted the add_cloudwatch_metricset branch May 1, 2019 22:07
@kaiyan-sheng
Copy link
Contributor Author

#11798 (comment) will be addressed in #12033

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
review Team:Integrations Label for the Integrations team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants